The popularity of sold state drives (SSDs) and exponential growth of network content has led to the emergence of all-non-volatile memory (NVM) storage systems, such as SSD arrays. The popularity of SSDs has also lead to the creation of a NVM Express® (NVMe) specification for SSDs connected to a Peripheral Component Interconnect (PCI) Express® bus referred to as “PCIe.” The merger of NVMe and network-based storage systems, referred to herein as NVMe Over Fabric (NVMEoF), leads to the concept of making PCIe-based NVMe SSDs accessible via a network fabric using a remote direct memory access (RDMA) transport.
An NVMEoF system includes one or more remote hosts connected to an NVMEoF system via a network interface card (NIC) through a network fabric, such as an Ethernet fabric. On the backend side, the NVMEoF system manages multiple SSDs, each of which can be shared with multiple remote hosts. The remote hosts issue read and write commands to the NVMEoF system over the network by layering the NVMe protocol over an RDMA protocol. The NVMEoF system forwards these read and write commands to the back end SSDs based on an internal mapping. The RDMA protocol allows data exchange without actively involving central processing units (CPUs) of the remote hosts, thereby reducing CPU overhead. One technique to exchange data between a remote host and the NVMEoF system is a store and forward mechanism between the NVMEoF system and the backend SSDs that provide the storage.
In a store and forward mechanism, all of the data for a particular operation (e.g., read operation) is transferred to memory of the NVMEoF system before being forwarded to the remote host. In case of a write operation, the entire chunk of data will be transferred from the remote host memory to the NVMEoF memory before being forwarded to the back end SSDs. Such a store and forward mechanism, however, requires the NVMEoF system to include a large amount of memory resources and increases the latency of any read or write operation as seen from the remote host. It is desirable to exchange data between remote hosts and an NVMEoF system using less memory resources and with low latency.